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WWC Review of the Report “The Role of Application 
Assistance and Information in College Decisions: Results from 

the H&R Block FAFSA Experiment ” 1 

The findings from this review do not reflect the full body of research evidence 
on providing financial aid assistance or information. 


What is this study about? 

In this study, the authors examined the impact of 
two interventions related to the Free Application 
for Federal Student Aid (FAFSA) on postsecondary 
outcomes of low- to moderate-income individuals. 
The two interventions were included (1) providing 
an estimate of need-based aid compared against 
tuition costs for nearby colleges and assistance in 
completing the FAFSA, and (2) only providing an 
estimate of need-based aid. The interventions were 
implemented by tax professionals in H&R Block 
offices in the Charlotte, North Carolina area and 
throughout the state of Ohio. The authors examined 
the impact of the interventions on four postsecond- 
ary outcomes: the likelihood of filing the FAFSA, col- 
lege enrollment, receipt of a Pell Grant, and retention 
in college after 2 years. 2 

Study authors measured the effects of the two FAFSA 
interventions by comparing the outcomes of three 
research groups created through random assignment 
based on the tax filers’ Social Security numbers: a 
FAFSA intervention group (which received an esti- 
mate of need-based aid and assistance in completing 
the FAFSA); an information-only intervention group 
(which only received an estimate of need-based aid); 
and a no-intervention comparison group. 

In total, almost 17,000 individuals were included in 
the analytic samples described in the study. The 
authors distinguished between three groups of 


participants when presenting results: (a) 17-year-old 
high school seniors and recent graduates who were 
dependent on their parents; (b) independent adults 
aged 24 to 30 years with no college experience; 
and (c) independent adults aged 24 to 30 years with 
some college, but no degree. 3 


Features of the Free Application for Federal 
Student Aid (FAFSA) Experiment 


The FAFSA is an eight-page, detailed application that 
must be completed to receive federal financial aid. 

Participants identified for this study were from 
households with low- to moderate-income (less 
than $45,000) who had received tax-preparation 
assistance at H&R Block and had a family member 
aged 15 to 30 years with no bachelor’s degree. 

For the FAFSA intervention group members, tax pro- 
fessionals first conducted a brief interview to collect 
information needed to complete the FAFSA. These 
professionals then used interview responses and tax 
return data to provide assistance with completing 
the FAFSA and to provide an estimate of the amount 
of need-based aid the participant would receive. 

For the information-only group, tax preparers used 
data from tax returns to provide an estimate of the 
amount of need-based aid they would receive, but 
assistance completing the FAFSA was not provided. 

Comparison group members were provided a bro- 
chure containing general information on college costs 
and financial aid, but received no direct help complet- 
ing the FAFSA and no personalized aid estimation. 
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What did the study find? 

The study found that the group of 17-year-old high 
school seniors and recent graduates (dependents) 
who received the FAFSA intervention were signifi- 
cantly more likely than the comparison group to 
submit the FAFSA (56% vs. 40%), attend college 
(42% vs. 34%), and enroll in college for 2 consecu- 
tive years (36% vs. 28%). 

Independent adults aged 24-30 with no prior college 
experience who received the FAFSA intervention 
were significantly more likely than the comparison 
group to submit the FAFSA (43% vs. 16%), and 
enroll in college (12% vs. 10%). 

Independent adults aged 24-30 with prior college 
experience who received the FAFSA intervention 
were significantly more likely than the comparison 
group to submit the FAFSA (52% vs. 32%). 

The study found no statistically significant differ- 
ences between the information-only group and the 
comparison group on any outcomes. 


WWC Rating 


The research described in this 
report meets WWC evidence 
standards with reservations 

Strengths: This study is a well-executed 
randomized controlled trial. 

Cautions: This study was based on a randomized 
controlled trial; however, impacts were presented for 
three distinct subgroups, and attrition for each of the 
subgroups cannot be calculated due to unknown 
subgroup information at baseline . 4 For this reason, 
this study was reviewed as a quasi-experimental 
design. All comparisons presented in this report 
meet WWC standards with reservations . 5 
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Appendix A: Study details 

Bettinger, E., Long, B., Oreopoulos, P., & Sanbonmatsu, L. (2012). The role of application assistance 
and information in college decisions: Results from the H&R Block FAFSA experiment. Quarterly 
Journal of Economics, 127(3), 1205-1242. 


Setting 

The FAFSA experiment was conducted in 156 H&R Block tax preparation offices located in the 


Charlotte, North Carolina area, as well as throughout the state of Ohio. 

Study sample 

The sample of participants came from households earning less than $45,000 a year with at 
least one household member between the ages of 15 and 30 without a bachelor’s degree. 
When more than one individual in the household was eligible, the independent adult clos- 
est to age 18 was selected. There were three samples of interest presented in the study, and 
assignment was determined randomly according to the participant’s Social Security number. 
The three groups were: (a) 17-year-old high school seniors and recent graduates who were 
still dependent on their parents (analysis sample sizes were 390 FAFSA, 80 information-only, 
and 398 comparison group members); (b) 24- to 30-year-old independent adults with no prior 
college experience (analysis sample sizes were 4,389 FAFSA, 722 information-only, and 4,1 17 
comparison group members); and (c) 24- to 30-year-old independent adults with some prior 
college experience (analysis sample sizes were 3,085 FAFSA, 517 information-only, and 3,044 
comparison group members). Each participant was offered $20. 

Intervention 

group 

Two intervention conditions, the FAFSA intervention and the information-only intervention, 
were implemented immediately after a family’s taxes were completed at a local H&R Block tax 
office. Participants in the FAFSA intervention group received personal assistance completing 
and filing the FAFSA through the use of tax software that completed two-thirds of the FAFSA 
based on their completed tax returns and an interview to obtain the remaining information. 
Potential financial aid amounts were then provided to the individual along with tuition esti- 
mates for four local colleges. Seventy percent of participants in this condition had their FAFSA 
completed by H&R Block. For those in the information-only intervention, potential financial aid 
amounts based on tax information were provided to individuals along with tuition estimates for 
four local colleges. Participants in both intervention groups also received information on the 
importance of college and a financial aid brochure. 

Comparison 

group 

Tax professionals provided comparison group families with information on the importance of 
college as well as a financial aid brochure. 

Outcomes and 
measurement 

The four outcomes included in this review were: (a) submitting the FAFSA within 1 year of the 
experiment; (b) enrolling in college within 1 year of the experiment; (c) receiving a Pell Grant 
within 1 year of the experiment; and (d) retention in college as measured by enrolling in col- 
lege in each of the 2 years following the experiment. For a more detailed description of these 
outcome measures, see Appendix B. 
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Support for 
implementation 

Reason for 
review 


Tax preparers were trained on the FAFSA experiment’s procedures. The tax preparers were 
then monitored closely at first in order to identify and correct the process when it was not car- 
ried out correctly. 

This study was identified for review by the WWC because it was partially supported by a grant 
to the National Bureau of Economic Research (Principal Investigator: Eric Bettinger) from the 
National Center for Education Research (NCER) at the Institute of Education Sciences (IES). 
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Appendix B: Outcome measures for each domain 


Assistance for college 

Filed FAFSA during first year following 
experiment 

FAFSA filing was based on data obtained from the U.S. Department of Education (ED) on the universe of FAFSA 
applications. This outcome was based on a 1 -year time window from the time of the experiment. 

Received Pell Grant while attending 
college during first year following 
experiment 

Receipt of a Pell Grant was based on data obtained from the U.S. ED on the universe of Pell Grant recipients. 
This outcome was based on a 1 -year time window from the time of the experiment. 

College attendance 

Enrolled in college during first year 
following experiment 

College enrollment was based on a combination of data from the Ohio Board of Regents (0BR) and the National 
Student Clearinghouse (NSC) on college enrollment. The NSC covers 92.3% of all student enrollments in North 
Carolina and Ohio. Enrollment was defined as enrolling in one of the schools listed in the OBR or the NSC within 
1 year following the experiment. 

Retention in college for 2 consecutive 
years following experiment 

College retention was based on a combination of data from the OBR and the NSC on college enrollment. The 
NSC covers 92.3% of all student enrollments in North Carolina and Ohio. Retention was defined as enrolling in 
any one of the schools listed in the OBR or the NSC for 2 consecutive years following the experiment. 


Table Notes: The authors identified these as primary outcomes of interest for specific populations. Enrolling in college was the primary outcome of interest for dependents and 
independent adults with no prior college experience, while receiving a Pell grant was the primary interest for independent adults with prior college experience. The authors also 
highlight retention for dependents and adults with prior college experience, although results for those with prior college experience are not included in the study. Twenty-two 
additional outcomes were examined in this study but were not included in this report because they were not identified as primary outcomes of interest by the study authors. They 
include: (1) attended public college; (2) attended private college; (3) attended 4-year college; (4) attended 2-year college; (5) attended full-time; (6) attended part-time; (7) attended 
in-state; (8) attended out-of-state; (9) total schedule amount of federal grants; (10) received federal student loan; (11) date of FAFSA filing 2008 conditional on filing; (12) attended 
college in second year after experiment, April 2009 to March 2010; (13) first entered college in second year after experiment, April 2009 to March 2010; (14) first entered college 
in third year after experiment, April 2010 to December 2010; (15) entered college in first, second, or third year after experiment, April 2010 to December 2010; (16) total years in 
college, April 2008 to December 201 1 ; (1 7) received Pell Grant in second year after experiment, April 2009 to March 201 0; (1 8) first received Pell Grant in second year after experi- 
ment, April 2009 to March 2010; (19) first received Pell Grant in third year after experiment, April 2010 to December 2010; (20) received Pell Grant in first, second, or third year 
after experiment, April 2010 to December 2010; (21) received Pell Grant for 2 consecutive years, April 2008 to December 2011; and (22) total years received Pell Grant, April 2008 
to December 2011. 
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Appendix C.1 : Study findings for each domain — FAFSA intervention compared to comparison 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Assistance for college 

Filed FAFSA during first year 
following experiment 

Dependents 

788 

participants 

0.56 

0.40 

0.16 

0.38 

+15 

<0.05 

Filed FAFSA during first year 
following experiment 

Independents, 
no prior college 

8,506 

participants 

0.43 

0.16 

0.27 

0.82 

+30 

<0.05 

Filed FAFSA during first year 
following experiment 

Independents, 
prior college 

6,129 

participants 

0.52 

0.32 

0.20 

0.49 

+19 

<0.05 

Received Pell Grant while 
attending college during first 
year following experiment 

Independents, 
prior college 

6,129 

participants 

0.23 

0.21 

0.02 

0.06 

+2 

>0.05 

Domain average of assistance for college 





0.44 

+17 

Statistically 


significant 


College attendance 


Enrolled in college during 
first year following 
experiment 

Dependents 

788 

participants 

0.42 

0.34 

0.08 

0.21 

+8 

<0.05 

Enrolled in college during 
first year following 
experiment 

Independents, 
no prior college 

8,506 

participants 

0.12 

0.10 

0.02 

0.10 

+4 

<0.05 

Retention in college for 2 
consecutive years following 
experiment 

Dependents 

788 

participants 

0.36 

0.28 

0.08 

0.22 

+9 

<0.05 

Domain average of college attendance 





0.18 

+7 

Statistically 

significant 


Table Notes: Positive results for mean difference, effect size, and improvement index favor the intervention group; negative results favor the comparison group. The effect size is 
a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) in an average student’s outcome that can 
be expected if the student is given the intervention. The improvement index is an alternate presentation of the effect size, reflecting the change in an average student’s percen- 
tile rank that can be expected if the student is given the intervention. The WWC-computed average effect size is a simple average rounded to two decimal places; the average 
improvement index is calculated from the average effect size. The statistical significance of the study’s domain average was determined by the WWC. 

Study Notes: Multiple comparison adjustments that were performed for outcomes using the same sample within the same domain aligned with the authors’ calculations. The 
p-values presented here were reported in the original study. The FAFSA intervention is characterized as having a statistically significant positive effect on both receiving assis- 
tance for college and college attendance because univariate statistical tests are reported for each outcome measure, the effect for at least one measure within the domain is 
positive and statistically significant, and no effects are negative and statistically significant. 
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Appendix C.2: Study findings for the assistance for college domain — information-only intervention 
compared to comparison 


Mean 

(standard deviation) WWC calculations 

Domain and Study Sample Intervention Comparison Mean Effect Improvement 


outcome measure 

sample 

size 

group 

group 

difference 

size 

index 

p-value 

Assistance for college 

Filed FAFSA during first year 
following experiment 

Dependents 

478 

participants 

0.37 

0.40 

-0.03 

-0.09 

-3 

>0.05 

Filed FAFSA during first year 
following experiment 

Independents, 
no prior college 

4,839 

participants 

0.14 

0.16 

-0.02 

-0.11 

-4 

>0.05 

Filed FAFSA during first year 
following experiment 

Independents, 
prior college 

3,561 

participants 

0.35 

0.32 

0.03 

0.07 

+3 

>0.05 

Domain average of assistance for college 





-0.04 

-0.02 

Not 

statistically 

significant 


Table Notes: Positive results for mean difference, effect size, and improvement index favor the intervention group; negative results favor the comparison group. The effect size is 
a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) in an average student’s outcome that can 
be expected if the student is given the intervention. The improvement index is an alternate presentation of the effect size, reflecting the change in an average student's percen- 
tile rank that can be expected if the student is given the intervention. The WWC-computed average effect size is a simple average rounded to two decimal places; the average 
improvement index is calculated from the average effect size. The statistical significance of the study’s domain average was determined by the WWC. 

Study Notes: No corrections for clustering or multiple comparisons were needed. The p-values presented here were reported in the original study. The comparisons for the 
information-only intervention condition required baseline adjustments due to baseline differences in gender and adjusted gross income that were larger than 0.05 standard devia- 
tions. Since the authors reported the results of an analysis that controls for these and other baseline characteristics in Table IV of the study, we report those comparisons for the 
information-only intervention condition. The information-only intervention is characterized as having an indeterminate effect on assistance for college because univariate statisti- 
cal tests are reported for each outcome measure, and none are statistically significant or substantively important. 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors]) to assess whether the study design meets WWC evidence standards. The review reports the WWC’s assessment of whether 
the study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the single study review protocol, version 2.0. This review is based on the final/published 
version of the study. In December 201 0, the WWC reviewed a working paper of this study that examined some of the same outcomes 
(the earlier review can be found here: http://ies.ed.gov/ncee/wwc/quickreviewsum.aspx?sid=142). The final study, which includes 
additional findings on longer-term outcomes, serves as the basis for this review, and replaces the initial assessment based on the 
working paper. 

2 There were 22 outcomes included in the study that are not described in this WWC report. See the table notes in Appendix B for more 
information. Because the FAFSA intervention included assistance in filing the FAFSA, the filing FAFSA outcome could be considered 
an overaligned outcome. However, the outcome is not excluded in this review, since completing the FAFSA is not a guaranteed out- 
come when receiving the intervention. 

3 Some of the comparisons of outcomes presented in the study are excluded from this review because the authors purposefully identi- 
fied primary comparisons of interest. For each of the intervention arms and subgroups of participants, the authors identified outcomes 
that would be most relevant. Specifically, on p. 21 of the published report, the authors state that their “main analysis is limited to a 
very small number of pre-specified questions: (1) Does FAFSA filing increase with FAFSA or information treatments? (2) Does FAFSA 
treatment increase college enrollment for dependents or independents with no prior college? and (3) Does FAFSA treatment increase 
financial aid receipt for those already going to college?” Based on this, the WWC focused the review on comparisons of all three 
participant subgroups and all three intervention conditions for the first question; we include only the FAFSA and comparison condi- 
tions and only the dependents and independents with no prior college for the second question; and we include only the FAFSA and 
comparison conditions and only the independents with prior college experience for the third question. Given the emphasis on college 
retention for the dependent sample in the study abstract, we include the retention outcome for dependents only. Finally, because the 
study authors included the information-only group solely for the purposes of detecting differences in FAFSA submission rates (see p. 
1219 of the published report), when examining impacts for the information-only group, we only present impacts on the FAFSA submis- 
sion outcome. 

4 According to WWC correspondence with the authors, there is insufficient information about the initial sample sizes (at the time of 
random assignment) for each of the three subgroups for which impacts are presented. This fact precludes our ability to assess attrition 
for each of the comparisons and results in the review of the study as a quasi-experimental design instead of a randomized controlled 
trial. It should be noted that attrition is low for the overall independent sample; however, impact analyses were not presented for the 
full sample of independents (only for the subgroups of independents with and without prior college experience). 

5 A working paper version of this study was previously reviewed as a quick review under earlier versions of the WWC standards. For 
that publication, the WWC determined that the contrasts focusing on the pooled independent adult sample receiving the FAFSA 
intervention met WWC standards without reservations. Using the current version of the WWC Evidence Standards (version 2.1) for the 
unpooled independent adult sample, it was necessary to assess the attrition levels of the analytic samples on which results are pre- 
sented, and in this study, there was insufficient information available to determine attrition levels for each of the two independent adult 
subsamples. As such, these comparisons were only eligible to meet WWC standards with reservations. In the same quick review, it 
was also found that the contrasts focusing on the information-only intervention for the pooled independent sample did not meet WWC 
evidence standards because the groups differed on previous college enrollment. Because the updated version of the study presents 
results by previous college enrollment, these baseline differences no longer exist. The authors further demonstrate equivalence for the 
information-only contrasts. Therefore, for this review, the information-only contrasts meet standards with reservations. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2013, November). 
WWC review of the report: The role of application assistance and information in college decisions: Results 
from the H&R Block FAFSA experiment. Retrieved from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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